Multimedia Content Understanding : Bringing Context to Content

In this paper, we propose a framework to extendsemantic labeling of images to video shot sequences and achieveefficient and semantic-aware spatiotemporal video segmentation.This task faces two major challenges, namely the temporal vari-ations within a video sequence which affect image segmentationand labeling, and the computational cost of region labeling.Guided by these limitations, we design a method where spatiotem-poral segmentation and object labeling are coupled to achievesemantic annotation of video shots. An internal graph structurethat describes both visual and semantic properties of image andvideo regions is adopted. The process of spatiotemporal semanticsegmentation is subdivided in two stages: Firstly, the video shot issplit into small block of frames. Spatiotemporal regions (volumes)are extracted and labeled individually within each block. Then,we iteratively merge consecutive blocks by a matching procedurewhich considers both semantic and visual properties. Results onreal video sequences show the potential of our approach.